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DoD  High  Perfonnance  Computing  Modernization  Program  (HPCMP) 

Abstract 

The  DoD  HPCMP  CREATE  program  is  developing  and  deploying  multi-physics 
High  Performance  Computing  (HPC)  software  applications  for  engineers  to  design  and  make 
accurate  predictions  of  the  perfonnance  of  military  air  craft,  ships,  ground  vehicles  and  radio 
frequency  antennas.  When  CREATE  started  (-2007),  there  was  no  commonly  recognized  set 
of  successful  software  project  management  practices  for  the  development  of  multi -physics, 
HPC  engineering  software.  Based  on  lessons-learned  from  the  HPC  and  computational 
engineering  community,  the  CREATE  leadership  developed  and  implemented  a  risk-based, 
practice-centered  strategy  to  organize  and  manage  the  highly-distributed  CREATE  program. 
This  approach  led  to  a  good  balance  between  ensuring  a  sufficiently  structured  workflow  and 
accountability  and  providing  the  flexibility  and  agility  necessary  to  create  new  sets  of 
engineering  tools  with  the  capabilities  needed  to  design  next-generation  weapon  systems. 

Introduction 

The  Department  of  Defense  (DoD)  High  Performance  Computing  Modernization 
Program  (HPCMP)  Computational  Research  and  Engineering  Acquisition  Tools  and 
Environments  (CREATE)  Program,  described  in  the  CREATE  program  Overview[l],  is 
developing  and  deploying  thirteen  physics-based  high  perfonnance  computing  applications 
to  DoD  engineers  to  make  accurate  predictions  of  the  perfonnance  of  military  aircraft,  ships, 
radio  frequency  antennas,  and  ground  vehicles.  These  predictions  are  designed  to  reduce  the 
risks,  costs,  and  time  to  develop  and  deploy  these  weapon  systems,  and  improve  their 
performance.  Using  the  CREATE  tools,  DoD  engineers  are  able  to  construct  and  test  virtual 
prototypes  of  weapon  systems.  This  enables  them  to  identify  and  fix  design  flaws  and 
performance  shortfalls  early  in  the  acquisition  process,  well  before  live  tests  are  possible  and 
“metal  has  been  cut.”  Experiences  in  industry  and  other  federal  agencies  demonstrated  that 
use  of  this  paradigm  can  reduce  costly  rework  and  result  in  reduced  risks,  costs,  and  time  to 
deployment  as  well  as  improve  product  perfonnance  and  quality[2,  3]. 

This  paper  describes  the  practices  that  were  developed  and  implemented  over  the  past 
nine  years  to  manage  the  key  CREATE  risks  that  confronted  the  CREATE  program.  The 
CREATE  tools  are  based  on  many  years  of  scientific  and  engineering  research  that  has  been 
instantiated  in  software  in  “research”  codes.  The  challenge  is  to  turn  this  knowledge  into 
engineering  software  that  can  confidently  be  used  by  engineers  who  didn’t  develop  the 
software.  The  risk  is  that  the  software  won’t  be  sufficiently  robust  and  usable  for  the 
engineering  community.  Many,  if  not  most,  such  projects  have  failed  in  the  past  e.g.  [4], 

Our  experience  and  the  recommendations  in  the  program  management  literature  e.g. 
[5]  are  that  all  successful  programs  have  three  simple  but  essential  elements:  1)  a  program 
vision  that  identifies  the  program  benefit  and  goals  that  define  the  requirements;  2)  a  strategy 
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that  is  expressed  in  plans  and  processes  for  execution;  and  3)  the  resources  required  to 
execute  the  program  including  stable  funding,  a  well-led  team  with  the  requisite  technical 
and  project  skill  set,  and  the  necessary  infrastructure  and  institutional  support.  Success 
requires  all  three  elements  be  adapted  to  fit  the  specific  program  environment  and  constraints 
(the  DoD  in  the  case  of  CREATE). 

These  CREATE  practices  are  based  on  a  set  of  key  principles  (Table  1)  that 
incorporate  these  key  elements  and  are  designed  to  reduce  those  risks.  These  principles  are 
based  on:  1)  the  authors’  collective  professional  experiences  and  “lessons  learned”  over  their 
careers;  2)  published  case  studies  of  similar  projects;  3)  the  software  project  management 
literature  and  training  courses  taken  by  the  authors,  and  4)  successful  practices  for 
commercial,  multi-national  product  development  and  adoption  supporting  30,000  software 
installations.  None  of  these  principles  are  surprising.  Many  of  them  are  in  the  standard 
literature[6]  or  appear  in  the  Agile  Manifesto  (  www.agilemanifesto.org).  However, 
relatively  few  science-based  software  projects  have  explicitly  implemented  them. 

Table  1:  Software  Project  Management  Principles  (Lessons-Leamed) 

•  Develop  a  compelling  and  credible  vision  and  be  able  to  communicate  it. 

—  Without  a  compelling,  credible  vision,  it  won ’t  be  possible  to  obtain  initial  funding  or 
to  defend  the  program 

•  Develop  a  long  term  strategic  plan  and  define  the  essential  processes  required  to  execute  it 

—  A  long  term  plan  is  essential  for  guiding  the  development  of  shorter-term,  more 
detailed  plans,  and  processes  are  required  to  execute  the  program 

•  Balance  the  need  for  an  agile  development  process  that  empowers  the  development  team 

with  the  need  for  accountability  and  an  organized  development  process 

—  The  development  teams  need  to  have  the  freedom  to  exercise  their  technical  judgment, 
but  also  need  to  produce  a  product 

•  Recognize  that  the  role  of  program  management  is  to  provide: 
o  Program  Leadership  beyond  merely  Program  Management 

o  Stable  funding,  hosting,  and  stakeholder  support  and  a  constructive  development  and 
deployment  environment, 

o  Guidance  for  solving  organizational  and  technical  problems,  and 
o  Support  to  solve  institutional  problems  the  development  team  is  unable  to  solve 

—  The  Program  Leadership  needs  to  shield  the  development  team  from  institutional 
turmoil  and  other  distractions  as  much  as  possible 

•  Recognize  that  the  role  of  the  development  team  is  to  provide  high  quality  software 

applications  within  schedule  and  budget 

—  The  development  team  needs  to  understand  its  responsibility  to  develop  and  deploy  the 
software 

•  Emphasize  the  central  and  essential  roles  of  the  development  team  and  its  leadership 

—  Teams  develop  software.  Organizations  and  processes  don  7  develop  software 

•  Implement  a  rigorous  verification  and  validation  program 

—  Without  a  good  V&V program,  your  codes  won  7  be  credible 

•  Identify  the  challenges  for  developing  and  deploying  the  software  within  your  organization 

and  customer  base 

—  The  Program  management  must  help  the  team  meet  those  challenges 
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CREATE  employs  thirteen  distributed  multi-disciplinary  non-collocated  teams 
centered  at  eight  separate  Department  of  Defense  (DoD)  sites  and  led  by  a  Program  Office 
located  at  the  HPCMP  office  in  Lorton,  VA.  The  CREATE  personnel  are  a  mixture  of 
government  civil  servants  and  defense  contractor  support  scientists,  engineers  and 
consultants.  These  personnel  are  located  at  29  dispersed  sites.  The  support  contractors  are 
drawn  from  25  separate  contractor  organizations.  CREATE’s  highly  distributed  development 
organization  illustrates  merely  one  of  the  five  “complexities”  (Table  2)  that  CREATE  has 
had  to  address  since  its  inception  in  2006. 

Table  2 — Five  Complexities  Specific  to  the  DoD 

•  Complex  program  management  challenges  (i.e.  development  of  new,  innovative,  risky, 
state-of-the-art  (and  beyond)  computational  technologies  in  DoD  organizations  that 
rigorously  adhere  to  DoD  program  and  project  management  processes). 

•  Complex  development  environment  (distributed,  multi-disciplinary  teams  across  multiple 
sites  and  embedded  in  Army,  Navy,  and  Air  Force  organizations). 

•  Complex  engineering  applications  (integrated  multi-scale,  multi-physics — e.g., 
computational  fluid  dynamics,  structural  dynamics,  turbo-machinery,  electromagnetics, 
etc.) 

•  Complex  computing  environments  (networks;  cyber  security;  rapidly  evolving  computer 
architectures,...) 

•  Complex  customer  organizations:  Army,  Navy,  Air  Force,  and  defense  industry 
acquisition  engineering  organizations;  the  DoD  research,  development,  testing  and 
evaluation  [RDT&E]  communities;  and  the  DoD  HPCMP. 

Establishing,  managing  and  leading  software  development  in  the  presence  of  these 
“complexities”  poses  many  Project  Management  challenges,  some  unique  to  the  DoD,  but 
many  that  are  common  to  all  distributed  physics-based  software  development  and 
deployment  projects.  This  paper  describes  the  Program  Management  approach  taken  by 
CREATE  Program  Office  to  develop  and  deploy  high-quality,  robust  engineering  HPC 
software  in  the  face  of  these  complexities. 

Sound  and  effective  software  engineering  practices  are  key  elements  for  success.  Those 
detailed  practices,  including  verification  and  validation,  code  development  methodologies, 
program  execution  issues,  etc.,  will  be  described  in  detail  in  subsequent  papers. 

CREATE  Program  Core  Risks 

The  initial  development  and  deployment  of  the  CREATE  program  was  planned  to  take 
twelve  years.  The  experience  of  projects  of  similar  complexity  and  scope  is  that  it  takes  that 
long  to  develop,  validate  and  deploy  the  software  to  the  point  that  the  tools  can  demonstrate 
sufficient  value  to  warrant  continued  support  [7,  8].  This  estimate  is  consistent  with  the 
history  of  the  CREATE  projects  (now  completing  their  eighth  year).  Successful  engineering 
software  applications  that  have  proved  their  value  can  have  lifetimes  of  30  to  40  years  (e.g. 
50  years  in  the  case  of  NASTRAN,  http://www.mscsoftware.com/product/msc-nastran  ),  so 
that  it  is  probable  that  the  CREATE  tools  can  have  a  long,  useful  life  if  their  development 
and  support  continues. 
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Each  of  the  five  “complexities”  of  CREATE  is  a  source  of  risk  and  uncertainty.  Software 
development  is  universally  a  risky  undertaking  and  failure  is  common  and  widespread.  Going 
back  decades,  numerous  authors  have  described  the  sources  of  software  development  risk 
(e.g.,  Boehm[9]  or  DeMarco[10]).  CREATE  is  no  different.  The  Software  Engineering 
Institute  (SEI)  published  a  risk  taxonomy  for  scientific  and  engineering  software[l  1]  that 
organizes  the  sources  of  risk  around  the: 

•  Software  development  cycle 

•  Development  environment 

•  Programmatic  environment 

The  CREATE  program  software  management  approach  included  many  insights  gained 
from  the  Department  of  Energy’s  ASCI  Program  [7]  and  other  studies  sponsored  by  the 
Defense  Advanced  Research  Projects  Agency  (DARPA)  High-Productivity  Computing 
Systems  Program  (HPCS)[12]. 

At  the  outset,  the  CREATE  program  identified  ten  core  Program-level  risks  (Table  3)  in 
the  DoD  environment  that  it  would  have  to  address.  Of  course,  the  software  development 
cycle  and  environment  also  present  risks,  but  this  paper  will  focus  on  programmatic  risk 
leaving  the  fonner  to  subsequent  papers.  We  have  found  that  programmatic  risks  are  the  most 
difficult  to  manage  because  their  sources  are  often  very  difficult  to  influence  from  within  the 
Program.  The  other  two  sources  of  risk  can  largely  be  mitigated  by  the  adoption  of  sound 
software  engineering  practices  tailored  for  projects  like  CREATE. 

The  specific  program  management  practices  described  in  this  paper  were  developed  by 
applying  the  Software  Program  Management  Principles  (Table  1)  to  manage  the 
programmatic  risks  described  below.  There  is  no  implied  priority  in  the  listing  below.  At 
various  times  of  the  life  of  CREATE,  numbers  4.  And  5.  have  been  paramount;  at  other  times 
8.  has  been  the  greatest  concern.  Increasingly,  9.  And  10.  are  important  concerns  as  the 
CREATE  products  mature. 

Table  3:  Ten  Core  Program-level  Risks  for  CREATE 

1.  Inability  to  meet  the  challenge  of  creating  and  inventing  new,  innovative  software 
technologies  within  the  existing  DoD  program  and  project  management  structure. 

2.  Loss  of  Credibility  and  Effectiveness  due  to  defects  or  insufficiently  accurate  models  in 
the  software  that  would  result  in  inaccurate  results 

3.  Inability  to  build  and  manage  software  development  teams  because  the  CREATE 
program  relies  on  sponsoring  development  teams  embedded  in  and  part  of  the  relevant 
DoD  customer  organizations. 

4.  Significant  losses  of  core  development  staff  and  their  corporate  knowledge  due  to  severe 
funding  reductions  and  other  institutional  turmoil 

5.  Inability  to  ensure  program  coordination  within  the  diverse  management  cultures — 
including  security  management — within  different  DoD  organizations. 

6.  Inability  to  manage  requirements  creep  and  relevancy  over  the  major  development  phases 
of  the  project 
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7.  Inability  to  anticipate  and  respond  to  the  impact  of  rapidly  changing  computational  and 
computer  technologies  (especially  rapidly  changing  computer  architectures  and 
environments). 

8.  Loss  of  DoD  stakeholder  and  sponsor  support  due  to  frequent  turnover  of  senior  DoD 
personnel. 

9.  Loss  of  Control  of  Intellectual  Property  Rights. 

10.  Inability  to  support  CREATE  software  users 

The  Management  of  Risk  through  Practices 

The  distributed  nature  of  the  sponsorship,  customer  base,  and  likely  workforce  of 
CREATE  -  directly  reflected  in  several  of  the  core  risks — posed  a  major  program 
management  challenge.  DoD  programs  in  the  past  have  utilized  “strong,  highly  centralized 
program  management  with  well  defined,  detailed  processes”  for  large,  complex  programs. 
CREATE  is  complex,  encompassing  distributed  teams  located  at  many  DoD  and  DoD 
contractor  locations  composed  of  ~  135  government  and  contractor  staff.  The  teams  face 
major  technical  challenges  to  build  and  deploy  the  software.  However  CREATE  is  much 
smaller  than  most  DoD  acquisition  programs  and  thus  is  unable  to  have  much  influence  on 
DoD  policies  and  procedures.  Our  experience  indicates  that  process-heavy  program 
management  approaches  have  been  less  successful  for  software  development  projects  like 
CREATE  than  practice-driven  approaches[4,  13].  The  imposition  of  a  strong  focus  on 
processes  runs  the  risk  of  alienating  and  disempowering  technical  software  development  staff 
who  are  essential  to  the  success  of  CREATE.  Moreover,  the  different  cultures  of  the  Army, 
Navy  and  Air  Force,  the  home  organizations  of  most  of  the  CREATE  personnel,  and  the 
highly  distributed  nature  of  the  development  teams  rendered  the  adoption  of  a  monolithic, 
prescriptive,  process-driven  management  approach  infeasible.  Instead,  we  adopted  practices 
modified  appropriately  to  be  effective  for  environment  of  each  development  team. 

The  practices  we  present  here  were  present,  if  only  implicitly,  in  many  of  the  successful 
components  of  the  ASCI  (DOE)  and  HPCS  (DARPA)  programs  cited  above.  They  are 
consistent  with  the  intent  of  milestone-based,  process-oriented  approaches  like  the  Software 
Engineering  Institute’s  Capability  Maturity  Model  [14],  [see  www.sei.cmu.edu/cmmi/] .  The 
CREATE  team  adopted  practices  synthesized  from  the  DoD  Federal  Acquisition  Regulations 
to  ensure  compliance  with  DoD  regulations.  The  remainder  of  this  paper  will  illustrate  the 
connection  between  the  practices  of  CREATE  and  the  risks  they  mitigate  for  the  DoD 
environment. 

Risk  1.  Inability  to  meet  the  challenge  of  creating  and  inventing  new,  innovative  software 
technologies  within  the  existing  DoD  program  management  structure. 

The  Department  of  Defense  is  a  very  large,  military  organization  (~  3  Million 
employees)  with  a  large  budget  (~$600B).  It  operates  by  requiring  adherence  to  a  uniform  set 
of  processes  and  regulations  that  apply  to  the  DoD  civilian  workforce  as  well  as  military  staff 
[15]. 

Somewhat  in  contrast,  the  development  of  new  technical  software  for  the  design  of 
highly  innovative  weapon  systems  by  the  DoD  acquisition  engineering  community  is  a 
“creative”  process.  It  involves  the  invention  of  something  that  is  new  and  innovative, 
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something  that  cannot  be  produced  by  following  a  rigid,  detailed,  repetitive  process.  Success 
requires  experimentation,  taking  risks  with  the  possibility  of  failure,  and  “learning  by  doing”. 
It  is  not  a  repeatable  process.  CREATE  developed  a  program  management  approach  that 
satisfies  DoD  needs,  but  is  able  to  provide  the  necessary  agility  and  flexibility  for  the 
development  teams  to  be  innovative  and  take  the  risks  necessary  to  succeed.  Many  DoD- 
recommended  management  practices  are  designed  for  very  large  programs,  and  are  not 
needed  for  the  management  of  relatively  small  code  development  teams.  We  found  that  many 
typical  DoD  processes  such  as:  highly  detailed,  multi-year  plans;  Earned  Value  Management; 
detailed  requirements  lists;  elaborate  requirements  tracing  methods;  detailed  oversight  with 
frequent  formal  project  reviews;  and  extensive  reporting  processes  are  not  applicable  or 
effective  for  the  CREATE  development  process.  Earned  Value  Management  (EVM),  for 
instance,  is  a  standard  practice  within  DoD.  To  be  successfully  applied,  EVM  depends  on 
timely  and  accurate  cost  accounting  and  level-of-effort  data.  For  CREATE,  it  was  not 
possible  to  collect  this  data  in  a  timely  way  across  the  Program  due  to  the  differences  in  the 
Service’s  accounting  systems.  The  EVM  process  itself  also  had  to  be  tailored  to  the  flexible 
execution  of  workflow  tasks  characteristic  of  agile  software  development  methods. 

Practice:  Adopt  a  program  management  approach  that: 

•  Encourages  the  use  of  agile  software  development  methodologies[6]  (e.g.  Scrum[16]) 

and  flexible  planning; 

•  Emphasizes  leadership  in  contrast  to  management; 

•  Supports  and  facilitates  the  success  of  the  development  teams  but  also  instills  a  sense 

of  accountability  and  encourages  an  organized  code  development  process. 

•  Resist  the  imposition  of  bureaucratic  processes  that  do  not  add  value  to  the  code 

development  process. 

Risk  2.  Loss  of  Credibility  and  Effectiveness  due  to  defects  or  insufficiently  accurate 
models  in  the  software  that  would  result  in  inaccurate  results 

The  purpose  of  the  CREATE  software  is  to  enable  DoD  engineers  to  accurately  predict  the 
performance  of  DoD  weapon  systems.  Errors  (either  defects  or  insufficiently  accurate  or 
inappropriate  models  or  misuse  of  the  tools)  that  lead  to  incorrect  results  destroy  the  value  of 
the  codes.  Incorrect  results  waste  resources,  discredit  the  CREATE  tools,  and  could 
ultimately  cause  injury  and  death  and  lead  to  the  demise  of  the  CREATE  Program. 

Practice:  Require  Extensive  Validation  and  Verification  (V&V)  Testing  of  the  CREATE 
tools  and  Quantification  of  the  Uncertainties  of  the  Code  Results 

An  extensive  and  effective  verification  and  validation  program  is  essential  for  ensuring 
that  the  tools  provide  accurate  answers.  The  CREATE  codes  employ  a  rigorous  verification 
and  validation  effort,  based  on  best  practices  identified  by  its  founders  in  programs  like 
ASCI[7]  and  HPCS[17].  Recently  the  National  Academy  of  Sciences  has  published 
guidelines  for  the  testing  of  scientific  and  engineering  software[18].  CREATE  testing 
practices  generally  already  confonned  to  these  guidelines.  The  CREATE  V&V  practices  and 
other  detailed  software  engineering  practices  will  be  described  in  a  subsequent  paper.  The 
CREATE  tools  develop  decision  data  for  acquisition  programs.  Accurate  estimates  for  the 
uncertainties  in  the  data  are  essential  for  making  sound  decisions  based  on  that  data. 


6 


Risk  3.  Inability  to  build  software  development  teams  because  the  CREATE  program  has 
no  authority  to  hire  staff. 

The  DoD  funded  the  CREATE  Program  but  didn’t  assign  any  personnel  slots  to  the 
program.  A  number  of  staffing  options  were  considered  including  giving  the  lead 
responsibility  to  a  private  contractor,  a  DOE  national  laboratory,  a  DoD  University  Affiliated 
Research  Center,  or  one  of  the  service  laboratories.  The  DoD  needed  to  retain  “ownership”  of 
the  software.  While  the  relevant  subject  matter  expertise  has  always  resided  in  the  DoD 
customer  organizations,  few  of  them  have  been  able  to  secure  sustained  funding  for  code 
development;  nor  did  many  of  them  have  much  experience  developing  high  quality  multi¬ 
physics-based  high  perfonnance  computing  engineering  software  that  is  usable,  sustainable, 
maintainable,  extensible  and  portable.  The  final  choice  was  for  the  HPCMP  to  build  code 
development  teams  around  subject  matter  experts  within  the  DoD  acquisition  engineering 
organizations  such  as  the  Naval  Surface  Warfare  Center,  Carderock  Division — the  Navy 
organization  responsible  for  ship  design  tools  and  assessments.  The  HPCMP  was  able  to 
assemble  a  team  of  senior  leaders  with  successful  experience  developing  computational 
engineering  software  that  was  able  to  provide  the  code  development  leadership  and  guidance 
to  the  service-led  development  teams.  CREATE  sponsored  the  teams  by  provided  funding, 
computers,  and  development  services,  and  active  leadership  and  management  of  each  team’s 
code  development  process  in  collaboration  with  the  relevant  service  organization.  This 
approach  has  worked  well. 

A  major  advantage  of  embedding  the  teams  in  the  customer  organizations  is  that 
transition  of  the  codes  to  the  customer  engineering  organizations  is  compellingly 
straightforward.  The  design  engineers  who  use  the  codes  are  part  of  the  organization  that 
developed  the  codes.  That  ensures  a  level  of  trust  and  ease  of  adoption  that  is  difficult  to 
obtain  for  externally  developed  codes.  It  eliminates  many  of  the  barriers  that  often  inhibit 
adoption  of  new  technologies  by  the  DoD  acquisition  community. 

Practice:  Identify  a  principal  developer  within  the  customer  organization  (in  this  case,  one 
of  the  Armed  Services)  around  whom  one  can  build  a  team. 

Each  of  the  technology  areas  targeted  by  CREATE  had  native  champions  within 
specific  customer  (Service)  organizations.  Moreover,  many  of  the  champions  had  been 
instrumental  in  the  initial  development  of  the  CREATE  program  proposal  and  were  familiar 
with  the  technical  and  management  challenges.  It  was  straight-forward  in  many  cases  to 
recruit  a  team  leader  acceptable  to  both  the  CREATE  Leadership  and  to  the  relevant  Service 
host  organization.  In  other  cases,  a  lengthy  search  process  was  necessary.  Our  experience  is 
that  the  selection  of  the  right  team  leader  is  the  most  important  decision  affecting  the  success 
of  the  project.  In  several  cases,  we  had  to  try  two  or  three  team  leaders  before  finding  the 
right  person.  Recruiting  a  team  within  a  customer  organization  to  develop  tools  for  which  the 
need  was  already  recognized  by  the  organization  turned  out  to  be  feasible  with  the  addition 
of  CREATE  funding  and  oversight.  A  team  leader  must  be  able  to  recruit  and  retain  talented 
team  members.  Long-tenn  engineering  software  development  projects  like  CREATE  are 
very  risky.  As  noted  before,  many  such  projects  have  not  had  successful  outcomes[19]. 
Prospective  CREATE  team  members  need  to  believe  that  there  is  enough  potential  for 
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success  to  invest  five  or  ten  years  of  their  careers  in  the  project.  Professional  and  personal 
respect  for  the  team  leader  is  essential  for  recruiting  team  members  to  take  that  risk. 

In  addition,  a  team  leader  who  is  a  valued  senior  staff  member  of  the  customer 
organization  has  the  professional  and  technical  trust  of  the  leadership  of  that  organization.  As 
noted  before,  this  greatly  facilitates  adoption  of  the  code  by  that  organization  and  similar 
organizations.  For  instance,  the  lead  developer  of  the  Ships  hydrodynamic  code  leads  the 
computational  fluid  dynamics  effort  at  the  Naval  Surface  Warfare  Center  responsible  for 
surface  ship  design. 

Practice:  Employ  lean  (5-15  person)  development  teams  led  by  technical  experts. 

While  it  might  appear  that  “multi-physics”  implies  large  teams,  CREATE  chose  to  use 
small,  distributed  teams.  In  our  experience  a  close-knit  team  founded  on  trust  and  able  to 
communicate  effectively  within  the  team  is  the  most  important  success  factor  for 
development  of  complex,  large-scale  physics-based  software.  Trust  and  communication  is 
most  easily  established  within  a  small  team  (5-15  people)  of  highly  competent  staff, 
especially  if  the  team  is  not  collocated  (as  is  the  case  for  almost  all  of  the  CREATE  Projects). 
Moreover,  large  development  teams  inevitably  require  more  management  overhead  and 
process  than  small  teams.  The  scarcity  of  staff  with  different  sets  of  required  highly 
specialized  technical  skills  necessitated  enlisting  team  members  across  two  or  three  sites,  or 
more. 

Sound  technical  judgment,  especially  for  the  team  leader,  has  been  a  crucial  success 
factor  for  the  achievement  of  CREATE’s  technical  goals.  A  good  team  leader  needs  a  solid 
grounding  in  the  relevant  science/engineering  fields  and  must  have  experience  in  and 
knowledge  of  computational  science  and  engineering.  The  leader  must  possess  effective 
interpersonal  and  leadership  skills  needed  to  organize,  lead,  and  inspire  a  team,  and  be  able  to 
manage  the  project  and  its  finances. 

It’s  impossible  to  overstate  the  importance  of  the  code  development  team  and  its  leader. 
The  team  develops  the  software.  The  major  task  of  a  software  development  organization  and 
its  management  is  to  support  the  team  and  the  team  leader.  The  quality  of  the  team  and  the 
team  lead  is  the  best  predictor  for  the  success  of  the  project.  With  the  right  team  leader  and 
the  right  team,  there  is  a  good  chance  of  success.  The  wrong  team  leader  dooms  the  project. 
As  Tom  DeMarco  puts  it  in  his  novel  on  Software  Project  Management  [20]: 

“Get  the  right  people. 

Match  them  to  the  right  jobs. 

Keep  them  motivated. 

Help  their  teams  to  jell  and  stay  jelled. 

(All  the  rest  is  administrivia.)” 

— T.  DeMarco [20] 

At  the  Program  Office  level,  CREATE  recruited  a  diverse  set  of  technical  expert 
consultants  to  augment  support  of  management  issues  like  software  engineering,  acquisition 
processes,  customer  development,  IP  management,  security  issues  such  as  export  control,  etc. 

Risk  4.  Significant  losses  of  core  development  staff  and  their  corporate  knowledge  due  to 
severe  funding  reductions 
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By  far  the  most  important  asset  of  the  CREATE  Program  is  its  development  staff.  It  takes 
a  long  time  and  a  lot  of  effort  to  recruit  a  good  team  and  a  lot  of  time  for  the  team  to  jell  to 
establish  the  trust  and  working  relationships  necessary  for  productive  code  development.  To 
maintain  steady  progress,  a  stable  code  development  team  and  development  environment  is 
essential.  That,  in  turn,  requires  a  stable  funding  base  over  multiple  years.  Large  funding 
oscillations  almost  always  depress  morale,  and  often  lead  to  high  turnover  rates  and 
pennanent  loss  of  essential  staff.  This  greatly  increases  the  likelihood  of  project  failure.  The 
external  market  for  talented,  highly  motivated  software  engineers  and  subject  matter 
engineers  with  computer  skills  is  very  competitive.  The  CREATE  staff  have  many  attractive 
career  options,  almost  always  at  higher  salaries  and  with  more  attractive  working  conditions. 
They  are  working  on  CREATE  because  they  enjoy  the  work,  and  have  the  opportunity  to 
make  a  major  contribution  to  the  success  of  DoD  acquisition  programs. 

From  the  viewpoint  of  the  development  staff,  significant  withdrawals  of  project  funding 
are  interpreted  as  a  lack  of  confidence  in  the  project  and  its  viability,  and  a  signal  they  should 
start  looking  for  other  employment.  Members  of  the  development  teams  make  a  commitment 
to  spend  years  working  on  a  project  that  will  take  years  to  show  progress.  They  expect  a 
reciprocal  commitment,  even  if  implicit,  that  they  will  be  supported  to  make  the  project  a 
success.  Nonetheless,  the  reality  is  that  funding  levels  do  fluctuate,  and  must  be 
accommodated  to  minimize  the  negative  impact  on  the  project.  CREATE  has  adopted  the 
following  practices  to  allow  it  to  react  thoughtfully  to  funding  fluctuations: 

Practice:  Ensure  that  funding  requirements  for  the  proposed  deliverables  are  defined  and 
planned  for  each  budget  year  during  the  project/product  life  cycle  and  are  adjusted 
throughout  the  year,  as  changes  occur.  Emphasize  the  importance  of  stable  funding  to 
HPCMP  and  CREATE  stakeholders  and  to  senior  DoD  leadership. 

The  goals  and  high  level  plan  for  project  deliverables  are  captured  in  a  long-tenn 
roadmap.  Every  year,  a  project  plan  for  that  year  is  developed  taking  into  account  the  goals 
for  product  deliverables  contained  in  the  long-term  roadmap,  new  requirements  generated  by 
the  stakeholders,  new  opportunities  due  to  advances  in  computational  technologies  (new 
solutions  techniques  and  algorithms),  and  the  funding  levels  available  for  that  year. 

Practice:  Publish  long-term  product  roadmaps,  and  update  as  necessary,  but  at  least 
annually  in  detail  for  the  near-term. 

Risk  5.  Inability  to  ensure  Program  Coordination  within  the  Diverse  Management 
Cultures — especially  Security  Management — within  Different  DoD  Organizations. 

As  a  Tri-Service  program,  CREATE  must  be  able  to  work  effectively  across  Service 
boundaries.  Each  Armed  Service  (e.g.  Navy)  has  its  own  rules,  processes  and  cultures  that 
vary  among  its  components.  The  variation  is  stronger  between  the  different  Armed  Services 
(e.g.  Navy  and  Army).  “Management  by  Walking-around,”  scheduled  short  meetings 
between  the  management  staff,  and  frequent,  but  very  short  meetings  among  the  development 
teams  are  highly  recommended  by  the  computer  software  development  literature.  However, 
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achieving  this  is  challenging  when  development  teams  are  not  collocated,  and  computer 
security  issues  inhibit  communication  and  data  exchange. 

In  addition,  network  and  system  security  controls  present  challenges  throughout  the 
CREATE  lifecycle  from  software  development  through  customer  usage.  The  extremely 
heterogeneous  security  enclaves  across  the  Services  inhibit  the  development  and  usage  of 
CREATE  software  due  to  policies  preventing  client  software  installation  and  restrictions  on 
network  ports  and  protocols.  Some  sites  do  not  allow  any  fonn  of  videoconferencing.  These 
impediments  have  been  mitigated  to  some  extent  by  using  browser-based  Software  as  a 
Service  (SaaS)  access  to  CREATE  development  repositories,  issue  tracking,  testing, 
documentation,  user  forums,  and  software  execution  resources  that  requires  no  client 
software  installation,  thus  ameliorating  many  security  concerns. 

On  several  occasions,  network  access  for  several  code  teams  was  shut  down  for 
extended  periods  (up  to  two  years  in  one  case)  due  to  external  security  and  management 
issues.  That  risk  required  mitigations  and  workarounds. 

Practice:  Strongly  encourage  good  communication  among  team  members,  especially 
among  non-collocated  teams.  Provide  high  quality  video  conference  and  teleconference 
capabilities  so  that  the  teams  can  hold  frequent  virtual  meetings. 

The  HPCMP  Defense  Research  Engineering  Network  (DREN)  group  in  the  HPCMP 
established  and  supports  a  high  definition  video  conferencing  capability  with  high  quality 
audio  that  has  greatly  facilitated  communication  among  and  within  the  development  teams 
and  with  the  CREATE  Program  Office.  It  allows  a  limited  amount  of  “Management  by 
Walking  Around,”  and  helps  keep  the  Program  Office  in  touch  with  the  teams. 

Participation  in  professional  society  meetings  is  strongly  encouraged  for  career  growth 
and  learning  about  the  latest  developments  in  the  relevant  profession,  Unfortunately,  the 
recent  DoD  restrictions  on  conference  attendance  strongly  limits  this. 

Practice:  Coordinate  and  share  access  to  CREATE  software  and  information  through  a 
data  server  that  supports  the  whole  CREATE  program. 

In  a  program  as  diverse  as  CREATE,  transparency  pays  big  dividends.  Successes  (and 
failures)  in  one  area  of  CREATE  can  benefit  others.  In  addition,  the  application  development 
lifecycle  requires  basic  services:  code  and  documentation  repositories,  forums,  issue 
tracking,  and  continuous  code  integration  and  testing.  Also,  debug  and  testing  of 
multithreaded  and  multi-process  (i.e.  OpenMP,  MPI)  code  necessitates  access  to  dedicated 
supercomputers,  storage,  and  interconnects. 

In  order  to  facilitate  these  development  services,  the  CREATE  program  has  established 
and  operates  a  community  site  with  virtual  machines  for  the  CREATE  development  and  user 
communities [21],  The  CREATE  community  site  provides  Multi-Factor  Authentication 
(MFA)  and  hierarchical  authorization  for  access  to  CREATE  software,  documentation, 
configuration  management,  agile  dashboards,  requirements  and  defect  tracking,  tutorials  and 
even  the  CREATE  user  community  through  the  user  forums. 

Practice:  Establish  a  method  for  allowing  users  to  use  the  CREATE  software  through  a 
browser  on  their  Army,  Navy  or  Air  Force  systems. 
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The  Army,  Navy,  and  Air  Force  limit  most  of  their  staff,  including  engineers,  to 
Windows-based  PCs  with  only  Microsoft  Office  and  a  browser.  This  effectively  prevents  the 
majority  of  potential  users  from  gaining  access  to  the  CREATE  software.  The  CREATE 
program  and  the  HPCMP  developed  a  web-based  portal  capability  that  allows  access  to  the 
HPCMP  systems  through  a  browser  and  with  Multi-Factor  Authentication  (MFA)  and 
encrypted  access.  Engineers  with  access  to  the  Anny,  Navy  or  Air  Force  networks  can 
single-sign  onto  the  HPCMP  supercomputers  through  their  browsers;  establish  a  remote 
desktop;  set  up  their  problem;  execute  jobs;  store  the  results;  visualize  and  analyze  the  results 
in-situ;  and  download  the  summary  conclusions,  including  graphics  and  videos.  The  HPC 
Portal  integrates  various  forms  of  help,  community  forums,  and  tutorials  with  the  use  of  the 
tools  themselves.  In  this  model,  the  CREATE  software  is  near  the  large  datasets  generated  by 
the  high  perfonnance  applications,  reducing  the  network  load  for  transferring  data  between 
the  client  workstation  and  remote  file  systems. 

Finally,  the  HPC  Portal  secures  access  via  several  mechanisms,  including  MFA  and 
application-specific  authorization.  Since  no  software  is  installed  on  the  user  workstation, 
vulnerabilities  and  system  updates  are  transparently  completed  on  the  servers.  Identity 
management  leverages  DoD  credentials  such  as  Common  Access  Cards  (CAC),  one-time 
password  (OTP),  and  OpenID.  Security  is  managed  within  the  DREN,  as  opposed  to  every 
desktop.  As  a  “Software  as  a  Service”  (SaaS)  implementation,  execute-only  privileges  can  be 
enforced.  The  HPC  portal  also  facilitates  access  by  defense  contractors  to  the  HPCMP 
computers  and  CREATE  tools.  It  allows  their  engineers  to  gain  secure  access  through  both 
their  internal  and  DOD  computer  networks. 


Risk  6.  Inability  to  manage  requirements  creep  and  relevancy  over  major  development 
phases  of  the  project  [-twelve  years]. 

The  DoD  is  competing  with  the  rest  of  the  world  to  rapidly  develop  new  and  more 
effective  military  technologies  in  a  fast  changing  world.  As  a  result,  the  requirements  for  the 
CREATE  tools  change  continually.  The  CREATE  program  adopted  three  practices  to  address 
the  risk  of  requirements  creep  and  potential  irrelevancy.  The  first  relates  to  the  customer 
focus  needed  to  ensure  relevancy: 

Practice:  Express  customer  requirements  as  “ use-cases ”  in  customer-oriented  language 
that  stakeholders,  customers,  and  developers  can  understand. 

CREATE  requirements  were  originally  defined  and  are  continually  reassessed  by  DoD 
Service  stakeholders  and  potential  customers  (users)  through  representation  on  the  CREATE 
Project  Boards  of  Directors  (one  for  each  Project:  Air  Vehicles,  Ships,  RF  Antennas,  Ground 
Vehicles,  and  Meshing  and  Geometry).  In  addition,  the  CREATE  teams  include  many 
seasoned  professionals  who  follow  the  world-wide  progress  in  militarily  relevant  science  and 
technology,  including  computational  science  and  technology. 

The  requirements  are  expressed  in  sets  of  use-cases  [22]  that  describe  the  targeted  uses  of 
the  CREATE  codes  in  customer  and  stakeholder  language.  Use-cases  are  scenarios  that 
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describe  at  a  minimum:  (1)  the  targeted  user,  (2)  the  users’  goal(s)  in  context,  (3)  minimal 
functionality  or  performance  targets,  and  (4)  main  success  scenario(s).  Use-cases  drive 
specifications  that  translate  use-cases  into  software  features  and  quality  attributes.  This  not 
only  aids  in  developing,  describing  and  testing  the  developed  capability,  it  also  eliminates  the 
need  for  extensive  burdensome  and  opaque  requirements  traceability  processes  and  reports. 

Practice:  Manage  code  development  with  a  workflow  culminating  in  at  least  one 
‘distinguished’  release,  or  ‘version’  each  fiscal  year. 

Reaching  ‘closure’  with  releases  addresses  one  of  the  main  failings  of  many  code 
development  efforts.  A  long  delay  between  fonnulating  requirements  and  demonstrating 
solutions  often  results  in  a  mismatch  between  the  two  and  is  typically  a  source  of 
requirements  creep.  While  a  detailed  development  process  (work  flow)  has  not  been  not 
prescribed  by  the  CREATE  Program  Office,  adaptations  of  Agile  methods  like  Scrum 
(www.scrumalliance.org)  that  include  milestones  (an  annual  release  is  a  milestone)  have 
been  self-adopted  by  all  of  the  CREATE  development  teams. 

Agile  work  flow  management  methods  promote  frequent  release.  Within  CREATE  code 
teams,  there  are  typically  incremental  releases  between  major  annual  releases.  These  are 
encouraged,  and  minimize  the  consequences  of  the  poor  project  management  practice  of 
using  corrective  maintenance  mechanisms  for  minor  adaptive  maintenance. 

Frequent  releases: 

•  Provide  CREATE  prototypes  for  customer  testing  and  input, 

•  Provide  an  annual  demonstration  of  progress, 

•  Allow  developers  to  reach  closure  on  incremental  capabilities,  and 

•  Help  to  mitigate  requirements  creep. 

Together  with  emphasizing  the  importance  of  the  team  leader,  annual  releases  is  one  of  the  most 
important  CREATE  program  management  practices. 


Practice:  Employ  pilot  projects  to  solicit  customer  reaction  and  input  to  feature  and 
attribute  implementations. 

Pilot  projects  provide  another  way  to  ensure  that  the  development  effort  aligns  with 
customer  needs.  Like  prototypes — a  feature  of  the  Spiral  Development  Method — pilots  allow 
developers  to  solicit  input  for  improvements  by  providing  users  early  access  to  the  codes.  But 
in  this  case,  early  access  is  provided  to  versions  that  are  “hardened”  for  customer  use,  not  just 
trial  implementations  characteristic  of  prototypes.  Pilots  help  ensure  that  the  customers  can 
determine  the  value  of  the  codes  in  their  own  workflows.  They  help  developers  improve  the 
match  of  the  code  with  customer  workflows,  thus  avoiding  a  common  failure  mode  due  to  a 
mismatch  of  the  code  and  the  customer  workflow.  Pilots  provide  valuable  feedback  that 
allows  the  CREATE  development  teams  to  address  customer  problems  before  the  general 
release  occurs. 
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Risk  7.  Inability  to  anticipate  and  respond  to  the  impact  of  rapidly  changing  computational 
and  computer  technologies  (especially  rapidly  changing  computer  architectures  and 
environments). 


Practice:  Rely  on  proven  computational  science  and  engineering  and  computational 
mathematics  technologies  to  satisfy  customer-defined  use-cases. 

Although  some  attributes  of  the  implementations  depend  on  evolving  technology 
(especially  regarding  HPC  hardware  architectures),  the  customer  use-cases  are  met  with 
proven  computational  methods  for  the  technical  problems  being  addressed  (e.g.  fluid 
dynamics,  structural  analysis,  ...)  that  do  not  require  research  breakthroughs.  While  the 
scaling  performance  at  the  margins  may  be  at  risk,  the  basic  functionality  is  not. 

Practice:  Ensure  that  the  CREATE  program  maintains  an  awareness  of  the  evolving  state 
of  the  art  in  high  performance  computing  and  its  implications  for  enhancing  the 
performance  of  the  CREATE  applications  and  keeping  its  computational  technologies 
modern. 

Computer  architectures  are  evolving  rapidly  and  methods  for  exploiting  these 
architectures  will  likely  require  significant  modernization  of  existing  codes  and  solution 
algorithms  [23].  The  CREATE  program  and  the  HCPMP  do  not  have  the  resources  to  meet 
this  challenge  on  their  own.  CREATE  has  been  preparing  itself  to  be  able  to  rapidly  adopt  the 
solutions  developed  by  the  software  divisions  of  the  major  computer  chip  vendors  and  the 
general  High  Perfonnance  Computing  (HPC)  community.  The  CREATE  codes  are  all  highly 
modular  so  that  the  work  required  to  modernize  the  codes  is  minimized.  In  addition,  senior 
HPCMP  and  CREATE  staff  interact  regularly  with  DOE  and  NSF  staff  and  other  research 
organizations,  and  monitor  progress  in  the  field.  CREATE  and  the  HPCMP  co-sponsor  pilot 
projects  with  groups  building  computational  mathematics  libraries  to  stay  abreast  of  recent 
developments  and  assess  their  impact  on  CREATE  and  DoD  HPC  codes. 

Risk  8.  Loss  of  DoD  stakeholder  and  sponsor  support  due  to  frequent  changes  of  senior 
DoD  personnel. 


Federal  programs  usually  have  stakeholders  consisting  of  both  sponsors  and  customers — 
sometimes  the  same  organization,  but  not  always.  The  personnel  in  these  two  communities 
change  frequently,  especially  senior  DoD  and  Service  leaders.  At  one  time  or  another,  these 
changes  have  jeopardized  the  success  and  even  the  existence  of  every  CREATE  code  team, 
and  a  few  times  have  threatened  the  existence  of  the  whole  CREATE  Program.  As  one  step 
to  cope  with  these  changes,  CREATE  leverages  its  Boards  of  Directors,  and  continually 
meets  with  and  informs  new  stakeholders  to  ensure  their  support. 

Practice:  Form  and  Convene  the  CREATE  project  Boards  of  Directors  (BoDs)  at  least 
annually  to  help  ensure  that  stakeholder  organizations  remain  engaged. 
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As  indicated  above,  each  CREATE  project  has  a  Board  of  Directors  composed  of  senior 
leaders  (primarily  Senior  Executive  Service  or  their  military  equivalents)  of  the  DoD 
stakeholder  communities  within  the  Services.  In  addition  to  the  role  of  ratifying  the  initial 
use-cases,  these  Boards: 

•  Provide  timely  advice  and  input  on  project  plans 

•  Provide  project  advocacy,  and  provide  Service  and  acquisition  community  input/views 

•  Provide  senior  customer  oversight 

•  Reaffirm  that  the  product  use-cases  remain  relevant,  or  propose  new  use-cases  (this  also 

helps  address  Risk  6) 

•  Ensure  that  new  Board  members  are  recruited  to  replace  those  who  are  transferred  to  new 

assignments  (so  that  contact  with  sponsors  and  customers  is  not  lost) 

Practice:  Continually  reach  out  to  new  senior  and  middle  level  members  of  the  DoD 
acquisition  engineering  community  (government  and  industry)  to  acquaint  them  with  the 
potential  of  CREATE  to  improve  acquisition  customer  outcomes.  Maintain  relationships 
with  those  who  supported  CREATE,  but  have  moved  to  new  responsibilities. 

There  must  be  an  ongoing  strategic  outreach  activity  due  to  the  frequent  turnover  of 
senior  DoD  civilian  leaders  and  military  staff  during  the  CREATE  life  cycle.  In  general, 
budgets  in  the  DoD  are  a  zero  sum  game.  Every  new  DoD  leader  has  new  ideas  that  are 
usually  require  starting  new  programs.  New  programs  require  funds  that  generally  have  to  be 
diverted  from  existing  programs.  It  is  thus  essential  to  be  able  to  convince  the  new  leadership 
that  your  program  is  more  valuable  than  the  competitors.  To  lose  your  program,  you  only 
have  to  lose  that  argument  once. 

Risk  9.  Loss  of  Control  of  Intellectual  Property  Rights, 

The  CREATE  tools  are  military  assets  designed  to  provide  the  DoD  with  a  military 
competitive  advantage.  If  the  DoD  cannot  maintain  control  of  the  distribution  of  the 
CREATE  tools,  that  military  competitive  advantage  is  at  risk. 

A  business  can  protect  its  intellectual  property  in  a  number  of  ways,  for  example,  with 
patents,  copyrights,  and  trademarks.  CREATE,  as  a  DoD  enterprise,  cannot  take  advantage  of 
patents  without  exposing  sensitive  infonnation  that  would  compromise  the  Government’s 
military  competitive  advantage.  Also,  Federal  employees  do  not  automatically  receive  a 
copyright  for  their  work  products.  To  cope  with  these  limitations,  CREATE  adopted  the 
following  three  practices. 

Practice:  Require  a  Standard  Software  Distribution  Agreement  (a  license  for  use). 

A  unifonn  distribution  (license)  agreement  has  three  important  benefits: 

•  Clearly  specified  restrictions  on  the  use  of  the  code  (such  as  not  selling,  sublicensing, 
transferring,  reverse  compiling,  or  reverse  engineering  the  code) 

•  Provides  a  warning  that  the  CREATE  software  and  associated  materials  are  export 
controlled  under  the  International  Traffic  in  Anns  Regulations  (ITAR) 
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•  Identifies  the  user  and  the  intended  use 


Practice:  Trademark  the  CREATE  tools  (protect  the  CREATE  brand). 

In  the  absence  of  patents  and  copyrights,  CREATE  has  adopted  the  practice  of 
trademarking  its  products.  This  practice  might  not  seem  necessary  for  software  tools  used 
primarily  within  the  DoD  and  its  contractors.  However  CREATE  applications  have  been 
licensed  to  commercial  companies  whose  business  is  not  entirely  confined  to  DoD  contracts. 
This  poses  a  greater  risk  that  CREATE  codes  could  be  misappropriated  or  that  results  from 
other  codes  or  unauthorized  CREATE  variants  could  be  misattributed  to  CREATE.  The  use 
of  a  trademark  identifies  the  CREATE  software  tools  as  government-owned  and  tested  and 
provides  a  basis  for  removal  of  the  CREATE  name  from  unauthorized  software. 

Practice:  Acquire  the  Necessary  Rights  in  Contracts  and  Licenses. 


Specifically  ensure  that  all  the  contracts  for  developing  CREATE  software  include  the 
Defense  Federal  Acquisition  Regulations  (DFARS)  clauses  that  grant  unrestricted 
distribution  rights  to  the  DoD. 

Like  most  modern  software,  CREATE  tools  make  use  of  3rd  party  applications  to 
generate  much  of  the  code  functionality.  These  applications  range  from  3ld  party  solvers  (for 
example,  linear  equation  solvers)  to  libraries  (example,  message  passing)  to  embedded 
components  (local  grid  refinement  routines).  The  use  of  third  party  components  by  the 
CREATE  software  development  teams  is  a  key  risk  area  which  requires  careful  selection  of 
the  components  not  only  for  their  technical  abilities  but  also  for  their  licensing  obligations.  A 
single  instance  of  the  insertion  of  code  into  CREATE  tools  without  review  and  approval  of 
the  validity  of  the  license  agreement  could  jeopardize  the  ability  to  distribute  the  application, 
cause  the  Government  to  lose  the  right  to  use  the  application,  and  subject  the  Government  to 
legal  action  to  recover  alleged  “damages”  for  violating  license  restrictions.  Consequently,  all 
vendor  licenses  are  reviewed  by  the  CREATE  legal  advisors  to  determine  if  distribution  of 
CREATE  will  be  inhibited  by  any  standard  license  restrictions,  copyrights,  or  patents.  If 
necessary,  new  licenses  are  negotiated  or  alternate  software  is  licensed  or  developed 
internally.  In  addition,  an  audit  is  perfonned  of  the  source  code  before  release  in  order  to 
verify  thoroughly  that  all  the  terms  of  the  respective  third  party  licenses  are  acknowledged 
before  releasing  derivative  or  extended  software.  This  is  standard  practice  for  the  commercial 
software  industry. 

The  use  of  contracted  software  development  support  personnel  for  the  CREATE  program 
poses  a  risk  of  not  having  unlimited  Government  rights  to  distribute  the  software.  If  the 
Government  pays  for  100%  of  the  software  development  effort,  CREATE  insists  on 
unlimited  use  rights;  for  software  development  contracts  where  the  software  has  only  been 
partially  funded  by  the  Government,  CREATE  receives  government  purpose  rights.  In  all 
cases  the  government  retains  exclusive,  irrevocable  rights  to  use  software  developed  by  a 
contractor  in  accordance  with  Defense  Federal  Acquisition  Regulations 
(www.farsite.hill.af.mil). 
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Risk  10.  Inability  to  support  the  CREATE  software  users 


User  support  is  a  key  requirement  for  success.  Users  must  have  a  good  working 
knowledge  of  the  strengths  and  weaknesses  of  the  software  tools,  and  be  skilled  users  of  the 
tools.  They  need  training  and  support.  Defects  must  be  reported  and  fixed.  Shortcomings  in 
code  usability  must  be  remedied.  Deficiencies  in  models  must  be  identified  and  corrected.  A 
responsive  user  support  organization  is  essential  for  sustained  user  adoption  and  effective  use 
of  the  CREATE  tools. 

Obtaining  funding  for  user  support  has  many  challenges [24],  Senior  decision  makers — 
who  control  funding — can  easily  understand  the  need  to  explicitly  fund  a  test  or  research 
facility  (e.g.  a  wind  tunnel)  but  do  not  as  readily  accept  the  need  for  sustained  funding  for 
user  support  for  computational  tools.  In  almost  all  cases,  our  experience  is  that  funding  for 
user  support  only  becomes  available  after  the  tools  have  convincingly  proven  their  value, 
which  can  take  a  long  time,  even  5  to  10  years. 

Practice:  Establish  initial  small-scale  pilot  projects  for  user  support  to  develop  effective 
methods  for  user  support  and  to  establish  the  utility  and  necessity  of  user  support. 


Summary 

These  practices  have  enabled  the  CREATE  program  to  successfully  manage  the  ten  core 
risks  associated  with  the  five  “complexities”  for  nearly  nine  fiscal  years  through  three  federal 
administrations,  a  major  shift  in  the  program  “home”  (from  the  Office  of  the  Secretary  of 
Defense  to  the  Army  Corps  of  Engineers)  that  led  to  major  changes  in  financial  and  contracting 
procedures,  and  continual  personnel  changes  and  reorganizations  within  sponsoring  and 
customer  organizations. 

While  all  of  the  practices  described  above  have  been  important  during  the  life  of  CREATE, 
some  summary  statements  can  be  made  about  them: 

•  Implement  all  of  the  Software  Management  Principles  in  Table  1.  Leaving  out  even 
one  can  jeopardize  the  success  of  a  program. 

•  Understand  the  complexities  of  your  organization  (e.g.  Table  2)  and  make  the  effort 
to  identify  key  risks  and  possible  mitigations  up  front 

•  Do  not  ignore  programmatic  risks  like  program  sponsorship  and  institutional  tunnoil 
just  because  they  seem  intractable 

•  Release  your  software  frequently  (at  least  annually)  to  garner  the  input  and  support  of 
your  customers;  the  releases  should  offer  something  of  substance 

•  Technically  competent  leadership  at  the  program  and  development  team  level  is  a 
crucial  success  factor  for  technical  software  development  teams 

•  Do  not  underestimate  the  possible  impact  of  an  ever  tightening  security  environment 
on  distributed  development  teams  and  customers 

•  Managing  IP  rights,  especially  in  today’s  world  of  open  source  software  components, 
is  critical  to  the  right  to  distribute  your  software 
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The  Software  Project  Management  Principles  (Table  1)  apply  to  almost  all  programs  that 
develop  physics-based  computational  engineering  software.  The  CREATE  practices  were 
specifically  derived  for  the  DoD  environment.  However,  we  believe  that  most  of  the 
CREATE  practices  will  be  useful  for  any  program  developing  and  deploying  computational 
engineering  software  with  distributed,  non-collocated  teams. 
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